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ABSTRACT 



The University of California, Davis, is transforming large 
general education courses in a cost-effective way that maintains high 
academic standards in the face of rising student enrollments. Two multi-year 
pilot projects have demonstrated the feasibility of using rich online 
learning environments to reduce the dependence on formal lectures. This 
resulted in more opportunity for interactions in small groups and reduced the 
constraints of lecture hall capacity on student enrollment. Evaluation of 10 
more classes distributed across disciplines by a multidisciplinary group of 
faculty and staff funded by the Mellon Foundation has three components: 
student learning, cost, and user profiling. This paper demonstrates the rich 
learning environments used for the prototype projects (involving an Oracle 
relational database, Cold Fusion middleware, a Web server, and a Web browser) 
and presents the extensive evaluation data on outcomes, including traditional 
examination data, student perceptions assessed in high-yield anonymous 
questionnaires, and objective data, on how students used the materials 
available. (Contains 15 references.) (AEF) 
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Introduction 

The addition of on-line supplements to courses may improve student learning although 
that has not always been shown but it does so at the cost of increased faculty time and 
institutional resources. How can information technology improve education in a cost- 
neutral way without compromising learning? 

At UC Davis, demand exceeds available space in many classes required for graduation, 
and many general education (GE) classes fall into this category. These are known as 
“impacted” courses. Enrollment has outgrown our classrooms although faculty and 
teaching assistant numbers are, to some extent, keeping pace with enrollment growth. 
Large classrooms cannot be constructed quickly enough to solve the over-enrollment 
problem. Building new classrooms is not a short-term option and is also expensive. 
Moreover, some have questioned the. pedagogical value of building ever-larger lecture 
halls. Some UCD classes already accept enrollments of 500, and most have enrollments 
of around 200, reflecting the large classrooms available on campus. 

Several prototype projects at Davis have explored alternatives to larger lecture classes 
and two of these have coalesced into a campuswide initiative funded jointly by the 
administration through the Provost’s office and the Andrew W. Mellon Foundation. The 
prototypes have shown, over a multi-year period, that on-line course materials can 
replace lectures without compromising learning and perhaps with enhanced learning for 
the majority of students. The paper will demonstrate the rich learning environments used 
for the prototypes (involving an Oracle relational database, Cold Fusion middleware, a 
web server and a web browser) and present extensive evaluation data on outcomes, 
including traditional examination data, student perceptions assessed in high-yield 
anonymous questionnaires, and objective data, on how students used the materials 
available. 

The initial stages of the project, including the prototype courses, the subsequent design of 
a larger study, and obtaining extramural support, required the cooperation of many units 
particularly the Academic Senate, the Office of the Vice-Provost for Undergraduate 
Education, the Teaching Resources Center, the Division of Information Technology, and 
the Office of the Vice-Provost for Infonnation and Educational Technology. Individual 
faculty members directed the initial stages and did most of the design work. 

The leadership now involves full-time academic senate faculty members from Biological 
Chemistry, Spanish, The Graduate School of Management, and Computer Science with 
other faculty and staff from the Teaching Resources Center, Human and Community 
Development, the Office of Planning and Budget, and the Instructional Technology and 
Digital Media Center. Participating instructors are from art history, anthropology, 
political science, agriculture and range science, food science and technology, agricultural 
resource economics, plant biology, psychology, and other areas. 



Strategies 

We are investigating two strategies that address the problem of static lecture hall size in 
the face of growing enrollments. In the first strategy, students are offered a choice of 
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taking the class on-line or in-person. Thus, with appropriate faculty and teaching 
assistants the class can grow beyond the size of the lecture hall. In the second strategy, 
part of the course is offered only on-line and the corresponding face-to-face lectures are 
discontinued. This reduces the use of the large lecture hall by this class, making that 
teaching space available to other courses. Both strategies are being implemented and 
evaluated in the current project, which involves 1 1 large undergraduate GE courses. 

The first strategy for reducing overcrowding has been tested in a prototype, which was 
very successful and which is part of the current study. In the first offering of the 
prototype, more than half the students in this first-year GE course chose the on-line 
version 1 . The second strategy mentioned above is part of the transformation of Physics 
undergraduate education undertaken over a number of years by Drs. Potter and Coleman , 
who have shown that this strategy can be successful at UC Davis. Another application of 
this approach is being planned for Chemistry. These strategies will both be investigated 
in the current study. 

The On-line Environment 

In the situation described here, the quality of the on-line experience is particularly 
crucial. The basis of comparison is the high quality of education now experienced by 
students resident at the Campus. We need a significant number of students to choose the 
on-line version of the course. Thus, we have focused primarily on delivering course 
content in a rich on-line environment. Course administration issues are handled by 
existing campus facilities on a commercial course management system. The learning 
environment we are using for many of the test courses can be viewed at 
http://leamh. ..ucdavis.edu/BCM41QA/ . 

The pilot course, BCM410A “Molecular and Cell Biology”, was used to develop a 
versatile content delivery system built on a database that is accessed by Cold Fusion 3 
scripts through a web server. The production version uses an Oracle database running on 
a Sun server and the Apache Web server and Cold Fusion server running on another Sun 
server. These two computers are housed in the campus Data Center, a dedicated, secure, 
computer facility with “7x 24” support. When the Web site becomes the sole source of 
essential course content, it must be reliable. Thus, any plan to replace (as opposed to only 
enhance) in-person lectures with on-line course content must include provision for very 
high reliability servers and network. 

The environment includes frames for graphics (still, animated, video), sound, text, and 
navigation. The premise is that students will view the graphics while listening to the 
sound. The purpose of the text - usually a transcript of the sound - is to allow the use of 
hyperlinks to a glossary, but some students also prefer to read the text as well as, or 
instead of, listening to the sound. The presence of sound, which contains all the essential 
information, makes the design accessible to blind students while the text makes it 
accessible to deaf students. 

The most straightforward way for an instructor to put a course into this environment is to 
have the in-person lectures video-taped and then provide the video-tapes and the visual 
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materials (PowerPoint presentations, slides, overheads, etc.) to a student or staff person 
who constructs learning modules, typically one per slide, by: 

1. building a JPEG file or animated Flash file from the provided graphic 

2. digitizing the sound from the videotape 

3. transcribing the videotape. 

The files for each learning module are then entered into the database using HTML forms, 
and the learning modules are organized into lectures. This can all be done without 
bothering the instructor but, of course, the instructor has to “proof-read” the product. In 
addition, it may be necessary to re-record some short sections of sound where inadvertent 
“mis-speaks” or “audio-typos” occur. Some instructors prefer to re-record the whole 
sound in a sound studio; others prefer the “live” quality of the original recording. 

Students like to hear the instructor rather than an unknown voice, because it makes the 
experience more personal. It is amusing for an instructor to attend a social function with 
the students’ spouses present because the spouses recognize the instructor’s voice. 

The use of relatively small learning modules provides the opportunity for sharing of 
modules as envisaged in the IMS project 4 and IEEE standards activity . Additional 
materials can be added and integrated into the learning environment, such as the glossary, 
self-test questions, quizzes and other assignments, tutorials, advanced topics, intelligent 
email, and discipline-specific features. On-line quizzes are useful for encouraging 
students to “keep up”. The system can release a quiz at a pre-determined time, email the 
students that it is available, and then grade the results and email the students with their 
individual grades and the grade distribution for the class. It can also release explanations 
of the questions when the quiz closes. 

The system has not been used for graded quizzes or examinations because of the need for 
proctoring and the need to maintain the grading process as similar as possible between 
the in-person and on-line person and on-line versions. At this point, the total enrollment 
in a class does not exceed the lecture hall capacity so that the examinations can be given 
to all enroll ees in person in the lecture hall, as usual. This also provides a fallback in case 
the on-line version is not working although, in practice, we are finding more migration 
from the in-person lecture to the Web than vice versa. So, the examinations can be given 
in-person in the lecture hall, as usual. However, if we find conditions in which the on-line 
version is a viable alternative to the in-person version, then the University will want to 
enroll students into the class beyond the capacity of the lecture hall. This will require 
alternative arrangements for the examinations, such as using non-lecture space or evening 
or weekend hours when additional lecture halls may be available. 



Results from the pilot studies 

In a 5-year study of one course, the course was offered as in-person lectures with 
computer supplements in years 1 and 2 and again in year 5. In years 3 and 4, the in- 
person lectures were discontinued and students studied from the web-based materials 
only. 
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Although this format did release a lecture hall, in years 3 and 4, the primary objective 
was to change the role of the instructor. Unlike the courses in the current study, this was a 
required first-year course for medical students who already had bachelor s degrees. In 
recent years, in our experience, such students have begun medical school with an 
increasing ability to leam factual material and reproduce it in examinations but a 
decreasing ability to use higher levels of understanding such as analysis, synthesis and 
application 6 . The result is good performance on internal examinations and the national 
USMLE, but a perceived lack of the basic science background when they enter the 
clinical years. Thus, students complain that their basic science studies are irrelevant to 
clinical medicine, and clinicians are frustrated by the students’ inability to integrate basic 
science into their clinical studies. 

The pilot course, BCM410A “Molecular and Cell Biology”, approached this by 
discontinuing the in-person lectures and requiring the students to leam this material from 
on-line and other resources, while introducing small-group case-based discussions jointly 
led by the basic science instructor and a clinician. The results of this successful 
experiment have been reported elsewhere 7 and are largely outside the scope of this paper. 
However, a side effect of the design was that the need for a large lecture hall 5 times a 
week was replaced by the need to find small discussion rooms 13 times a week, which 
has clear implications for space planning. The issue of most relevance here is the success 
of student learning on-line. The small groups may have contributed an improvement in 
motivation, but they were designed to develop students’ higher order thinking skills, in a 
clinical context. The material discussed by the small groups was not closely tied to the 
course content (much to the dismay of the department 8 ). 

In years 1,2, and 5 (in-person lectures), about 30% of the students received A grades 
while in years 3 and 4 (on-line content presentation) about 50% received A grades 
(Figure 1). While this is not a properly controlled blind study it shows that, in the rich on- 
line learning environment employed, student learning at least didn’t suffer compared with 
the in-person version of the course and may well have improved. 

Meta-analysis of a large number of comparisons of on-line with in^person content 
presentation results in the “no significant difference” phenomenon . However, this 
meta-analysis may be burdened by methodological shortcomings. For example, many of 
the papers and research cited in the analysis themselves cross-reference (i.e., cite similar 
research and reference each other). Consequently, the amount of research that reaches 
the “no significant difference” conclusion may be inflated. Moreover, much of the 
research does not control for extraneous variables, or use randomly selected subjects. 
Also, the analyzed research sometimes lacks the use of valid and reliable instruments and 
does not always control for “reactive effects” (e.g., the novelty effect, which describes 
increased student motivation and interest due to using something different) 

Nevertheless, similar to our pilot course, the research does show that for the particular 
students involved, the online approach was successful. 

The indication of better learning in the pilot study with BCM410A may be due to the use 
of a rich learning environment, unlike many of the studies included in the meta-analysis, 
and/or the fact that only the course content was on-line while office hours and other 
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student contact occurred in person. This rich learning environment 10 is being used as the 
basis for most of the 1 1 other courses in the present study. 

Significant insight into the students’ views of the on-line content presentation were 
obtained by textual analysis of students’ free text responses to the following questions in 
an anonymous questionnaire returned by all the students: “What did you like about the 
virtual lectures” (Figure 2); “How could the virtual lectures be improved?” (Figure 3). 
From this analysis and quantitative answers to related questions we learned that a rich 
web-based learning environment which is the most popular way for these students to 
learn, sharp contrast to textbooks, in-person office hours, and other materials . These 
students did not, however, have the opportunity to compare directly the in-person lectures 
with the on-line content presentation. 

The other pilot course was an undergraduate general education course, like the other 
courses in the current study. This course, BIS 10 “General Biology”, was offered both 
on-line and in-person in the summer of 1999 and 2000. Students chose either the on-line 
version or the version with in-person lectures. In both versions of the course, assignments 
and required asynchronous discussions were carried out on-line using email. In both 
versions, the examinations were in-person. In this case, the lectures were text-based, 
without sound, but did include graphics, including some animations and simulations, and 
self-test quizzes. The examination data showed no significant difference in learning 
between the on-line content presentation and the in-person lectures. 

Both pilots showed excellent student acceptance of the on-line format and learning was 
either the same or better with the on-line format. These results provide the justification 
for the larger study that is now in progress. 

Cost analysis 

Analysis of costs was not part of the pilot studies. Significant costs were incurred, mostly 
faculty time but also some costs for student assistance and for hardware. The current 
study has cost analysis added as a major evaluation component. Any claim for a scalable 
solution to campus problems must include costs analysis as a key component. In this 
case, the alternatives to the proposed solution to over-crowding of lecture halls include: 
increased time to degree; renting of lecture space off campus; use of lecture halls on 
Sunday and/or outside the current 7:30 AM to 9:00 PM times; reduction of general 
education requirements. Reduced enrollments are politically impossible and building new 
classroom space is not a short-term solution. What are the relative costs of these 
alternatives? The current study will address the relative costs of the use of on-line content 
presentation in comparison with in-person lectures. 

The specific aims of the cost analysis are twofold: 

1) to estimate the differences in costs between the on-line and traditional lecture 
course offerings, and 

2) to develop a differential costs model for further testing and refinement based on 
the data collected for the ten courses being studied. 
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The analysis will focus on differences between on-line and traditional lecture courses 
without concern for assigning costs to other activities that should not differ (e.g., 
research). Activities that might differ between the traditional and on-line components are 
faculty time spent on developing course materials, preparing the course for on-line 
delivery, giving lectures; and other teaching personnel time (e.g., time spent by Teaching 
Assistants). 

Many of the costs will be joint, that is, costs incurred for both traditional and on-line 
courses. Faculty probably will spend time developing course materials in both modes. 

Joint costs will be dropped when computing differential costs. Using data from university 
records 12 , we will obtain estimates of space, equipment, and infrastructure costs that 
differentiate the teaching modes. 

We will compute indirect as well as direct costs. Indirect costs are associated with 
teaching but not with particular course offerings, for example, attending seminars to 
improve teaching skills. We predict some differences in indirect costs between the two 

modes. 

A key component of cost is instructor time. We developed a comprehensive, easy-to-use 
instructor time log by asking seven instructors to generate a list of all their activities with 
regard to offering a course. Additional items were gleaned from a campus-wide survey . 
The resulting list of activities was reviewed and sorted into logical categories of course- 
related tasks. The categories were subsequently checked to ascertain that they were 
mutually exclusive and comprehensive. Clear definitions for each category were 
constructed and checked for reliability. Following pilot testing, the log was posted online, 
to be filled out daily by instructors who are being prompted by an automatic e-mail 
message, which simply says: 

Please go to the following URL to update your confidential timesheet for 
the Mellon project: 

htm://cloud\'bav.uedavis.edu/Mellon/TimeShee t.cfm?Name=Fatk 

Thanks for helping us collect good contemporaneous data on faculty time. 

The URL brings up a Web form where instructors enter the time spent in three columns, 
“On-Line”, “Traditional”, and “Joint”, which reflect which version of the course the time 
was spent on. The time is divided into the following 12 categories. 

Direct Time 

1 . Working with project evaluators: 

Course evaluation for project evaluators , meeting with project evaluators, 
keeping the time log. 

2. Planning the course, developing materials, developing lecture content: 
Developing the course syllabus 

Reviewing textbooks and other materials 

Creating the course structure and content - writing lectures, making outlines, 

preparing assignments and class demonstrations 

Selecting topics and creating assignments for lab discussions 
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Organizing reading materials, duplicating readings, dealing with copyright 
requirements 

Locating, duplicating, and organizing media (e.g., scans, photocopies, slides, 
overheads), previewing videos/films 

Preparing web pages (except for online delivery which is category 3 below). 

3. Preparing the course for online delivery, reviewing materials for online delivery: 
(Course development) Constructing online teaching/learning modules by 
converting lectures to web format. 

4. Delivering the course in a particular term. Preparing for lectures, delivering 
lectures, dealing with problems with the online and traditional delivery of course 
content to students. 

Classroom tasks - obtaining equipment, setting up, immediate preparation for 
lecture/demos, in-class activities - lecture, facilitation, demonstration. Electronic 
tasks - consultation, server-related tasks, maintaining website. 

5. Interacting with students outside of class. 

Student contact - office hours, telephone, e-mail, after-class lingering, one-to-one 
sessions, special study sessions, troubleshooting, counseling, tutoring; managing 
student groups. 

6. Evaluating student performance. Preparing and grading examinations, grading 
papers and projects. Assigning grades. 

Constructing, typing, coding, administering, grading - exams/tests/quizzes, 
makeup exams, assignments, papers, projects, extra credit assignments 
Record-keeping - grade sheet, tracking assignments, printing, posting, submitting 
grades 

Resolving - incompletes, medical excuses, emergencies 

7. Training and supervising TAs and other assistants. 

TA contact - TA and/or Reader orientation, training, supervising, formal and 
informal meetings, e-mail, phone contact 

8. Other (please specify). 



Indirect Time: 



(i. Attending seminars, reading, collecting materials useful for teaching. 

10. Interacting with students that cannot be identified with a particular course 
offering. 

11. Planning and developing materials for future offerings of the course. 

12. Other (please specify). 

Expenditures: 

13. Please describe briefly in box belcw and put $ amounts in the boxes to the right. 
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When the form is submitted, the data is entered into an Excel spreadsheet, together with 
the date of submission. The spreadsheets, one for each instructor, can be easily 
downloaded by a project evaluator for a quick check of progress. At the end of each 
course, the data will be imported into a program that is suitable for more detailed 
statistical analysis. 

In comparing costs, we must also recognize that the cost of on-line courses may be 
initially inflated due to start-up costs. Since the traditional lecture method is well 
established, it likely requires less time and training to develop and execute. In addition, 
the on-line courses cannot take advantage of economies of scales, since so few are 
currently offered. With time, the on-line courses will likely become more cost effective 
and demonstrate increasingly greater cost advantage compared with the in-person lecture. 
While the cost of developing the on-line courses is obviously important, we must be 
aware of short term versus long term costs of using a new learning format. 

Conclusion 

As David Brown has said: “The jury is in! On-line learning works!” 14 . We agree, 
provided the on-line learning environment is sufficiently rich, and we have data to 
support this statement at our own institution. The question now is: “How do we use it?” 

Clearly on-line learning can be used by us and by others to reach students who are not 
resident on our campuses and we are collaborating with our Extension people in this area. 
However, we are directly interested in large-scale problems on campus where on-line 
learning can be at least part of the solution. Large-scale implies both scalability of the 
application framework used for on-line learning and cost-effectiveness. We have 
identified a particular acute problem on our campus but our data and experience will be 
valuable to everyone who needs to use information technology to improve education in a 
cost-effective way. 
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Appendices 



Figure 1. Student grades improved with on-line learning. 

Student grades for years 1995 through 1999; in 1997 and 1998, the course content was 
provided through the Web without the in-person lectures. In the other years, in-person 
lectures were given. 
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Figure 2. Textual analysis of students’ free text responses to “What 
did you like about the virtual lectures?” 
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Figure 3. Textual analysis of students’ free text responses to “How 
should the virtual lectures be improved?” 
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